Skip to content

[fix](preagg) Enable pre-aggregation for multi-argument aggregate functions with all-key distinct inputs#65846

Open
starocean999 wants to merge 4 commits into
apache:masterfrom
starocean999:master_0721
Open

[fix](preagg) Enable pre-aggregation for multi-argument aggregate functions with all-key distinct inputs#65846
starocean999 wants to merge 4 commits into
apache:masterfrom
starocean999:master_0721

Conversation

@starocean999

@starocean999 starocean999 commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #48502

Problem Summary:
When an aggregate function has more than one child expression (e.g., count(distinct k6, v7)), the pre-aggregation status check previously only handled the cases of zero children or exactly one child. Multi-child aggregate functions fell through to the else branch that attempted to process them via checkAggWithKeyAndValueSlots, which is designed for single aggregate functions with complex expressions like IF/CASE WHEN — not for genuinely multi-parameter aggregate functions. This could lead to incorrect pre-aggregation decisions for functions like COUNT(DISTINCT col1, col2).

This fix explicitly only enable pre-aggregation for multi-argument aggregate functions with all-key distinct inputs, and reject other multi-argument aggregate functions.

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@starocean999

Copy link
Copy Markdown
Contributor Author

/review

@starocean999

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29606 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6a34cab3c002ae3c9edf1133ff1a9b492da4b95e, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17680	4032	4022	4022
q2	2011	324	205	205
q3	10333	1435	848	848
q4	4679	472	343	343
q5	7510	859	568	568
q6	184	172	137	137
q7	752	830	616	616
q8	9339	1665	1555	1555
q9	5565	4393	4378	4378
q10	6773	1768	1499	1499
q11	504	352	330	330
q12	759	580	469	469
q13	18089	3416	2756	2756
q14	265	258	249	249
q15	q16	787	777	703	703
q17	971	957	1055	957
q18	7030	5692	5610	5610
q19	1273	1239	1003	1003
q20	768	682	577	577
q21	5904	2633	2486	2486
q22	423	344	295	295
Total cold run time: 101599 ms
Total hot run time: 29606 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4373	4301	4312	4301
q2	282	320	215	215
q3	4518	4948	4419	4419
q4	2060	2147	1373	1373
q5	4413	4293	4439	4293
q6	232	176	124	124
q7	1722	1879	1831	1831
q8	2600	2182	2176	2176
q9	7968	8195	7750	7750
q10	4685	4747	4233	4233
q11	553	422	390	390
q12	745	749	527	527
q13	3272	3663	3002	3002
q14	299	303	281	281
q15	q16	698	745	639	639
q17	1348	1312	1311	1311
q18	8084	7278	7304	7278
q19	1217	1150	1101	1101
q20	2215	2228	1937	1937
q21	5232	4577	4379	4379
q22	525	474	406	406
Total cold run time: 57041 ms
Total hot run time: 51966 ms

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review summary

Requested changes for one P2 performance regression: the new blanket multi-child gate fixes the unsafe mixed key/value COUNT DISTINCT case, but it also disables valid partial/parallel scanning for all-key multi-column COUNT DISTINCT. The inline comment contains the concrete plan, impact, and fix/test boundary.

Critical checkpoint conclusions:

  • Goal and correctness: count(distinct k6, v7) on an aggregate-key table now correctly gets PREAGGREGATION OFF, preventing partial SUM states from changing the DISTINCT tuples. The implementation is broader than required and regresses the safe count(distinct k5, k6) plan.
  • Scope and focus: the two-file patch is mechanically small and readable, but children().size() > 1 is not the smallest semantic predicate; supported multi-child COUNT DISTINCT arguments need per-scan validation.
  • Parallel paths and performance: both Nereids SET_PREAGG_STATUS registrations use this rule. For AGG_KEYS/UNIQUE-MOR scans, the unnecessary OFF decision requires a full storage merge and can remove an otherwise eligible scan from the BE parallel-scan path. DUP_KEYS and UNIQUE-MOW/MOR-as-DUP bypasses are unchanged.
  • Testing: the added EXPLAIN is a valid before-fail/after-pass trigger for the mixed key/value bug, but notContains ... PREAGGREGATION: ON does not prove the intended OFF reason and there is no positive all-key boundary. No result file is involved.
  • Compatibility and FE/BE propagation: no function symbol, storage format, configuration, or thrift field changes are introduced; the existing preaggregation flag is forwarded unchanged.
  • Concurrency, lifecycle, persistence, transactions, and writes: none are introduced or modified by this patch.
  • Error handling and observability: the new branch fails closed and EXPLAIN exposes the OFF reason; no additional log or metric is needed.
  • Other paths: aliases, nested aggregates, joins/multiple scans, nullable/duplicate keys, expression shapes, and later DISTINCT splitting were checked. No additional independent issue was substantiated.

User focus: no additional review focus was provided.

Validation: static review only. The review environment explicitly prohibits builds and test execution, so the added regression case was inspected but not run.

Review completion: every changed file and candidate was swept, deduplicated, and resolved; normal full-review and risk-focused agents converged within the three-round cap.

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 178944 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6a34cab3c002ae3c9edf1133ff1a9b492da4b95e, data reload: false

query5	4312	647	476	476
query6	465	239	207	207
query7	4845	576	330	330
query8	332	184	171	171
query9	8778	4070	4135	4070
query10	449	356	300	300
query11	5891	2327	2140	2140
query12	161	101	112	101
query13	1268	595	432	432
query14	6195	5338	4866	4866
query14_1	4258	4262	4214	4214
query15	207	201	186	186
query16	1001	493	482	482
query17	963	719	598	598
query18	2439	480	360	360
query19	214	191	152	152
query20	113	107	103	103
query21	240	158	137	137
query22	13620	13605	13340	13340
query23	17516	16541	16240	16240
query23_1	16234	16249	16256	16249
query24	7509	1744	1295	1295
query24_1	1301	1276	1269	1269
query25	583	469	409	409
query26	1336	363	213	213
query27	2595	587	373	373
query28	4487	1990	1971	1971
query29	1093	631	534	534
query30	354	272	230	230
query31	1118	1095	980	980
query32	114	65	61	61
query33	536	327	267	267
query34	1212	1145	649	649
query35	764	775	677	677
query36	1201	1165	1040	1040
query37	156	121	110	110
query38	1885	1707	1671	1671
query39	900	882	852	852
query39_1	836	853	862	853
query40	292	157	140	140
query41	64	66	64	64
query42	96	92	91	91
query43	324	348	289	289
query44	1448	788	765	765
query45	194	186	182	182
query46	1039	1192	732	732
query47	2088	2117	2006	2006
query48	395	403	288	288
query49	582	411	310	310
query50	1055	418	339	339
query51	11007	11080	11005	11005
query52	87	90	76	76
query53	269	290	203	203
query54	281	235	217	217
query55	79	70	69	69
query56	330	280	271	271
query57	1325	1307	1188	1188
query58	283	261	252	252
query59	1580	1639	1410	1410
query60	325	261	241	241
query61	148	145	147	145
query62	548	495	433	433
query63	244	209	201	201
query64	2851	1022	865	865
query65	4723	4622	4660	4622
query66	1825	529	396	396
query67	29293	29315	29156	29156
query68	3191	1647	1035	1035
query69	421	305	276	276
query70	1060	938	958	938
query71	353	332	311	311
query72	3034	2647	2365	2365
query73	809	722	445	445
query74	5079	4933	4698	4698
query75	2533	2506	2161	2161
query76	2335	1154	767	767
query77	350	373	285	285
query78	11907	11780	11360	11360
query79	1385	1149	739	739
query80	1167	555	453	453
query81	511	338	288	288
query82	561	159	129	129
query83	393	322	301	301
query84	330	157	134	134
query85	979	645	525	525
query86	413	295	275	275
query87	1820	1814	1760	1760
query88	3718	2808	2753	2753
query89	440	376	338	338
query90	1823	209	206	206
query91	212	196	163	163
query92	64	59	56	56
query93	1586	1510	971	971
query94	635	345	339	339
query95	797	533	466	466
query96	1102	801	353	353
query97	2607	2617	2501	2501
query98	216	207	201	201
query99	1095	1119	1015	1015
Total cold run time: 263414 ms
Total hot run time: 178944 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 25.13 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6a34cab3c002ae3c9edf1133ff1a9b492da4b95e, data reload: false

query1	0.00	0.00	0.00
query2	0.10	0.05	0.04
query3	0.25	0.14	0.14
query4	1.62	0.15	0.15
query5	0.24	0.27	0.23
query6	1.26	1.09	1.06
query7	0.04	0.01	0.01
query8	0.05	0.04	0.04
query9	0.39	0.31	0.32
query10	0.56	0.55	0.55
query11	0.20	0.13	0.14
query12	0.18	0.13	0.14
query13	0.48	0.47	0.48
query14	1.03	1.01	1.01
query15	0.61	0.58	0.59
query16	0.33	0.32	0.31
query17	1.10	1.12	1.12
query18	0.24	0.21	0.21
query19	2.04	1.94	1.93
query20	0.02	0.01	0.01
query21	15.43	0.22	0.13
query22	4.80	0.05	0.05
query23	16.12	0.29	0.12
query24	3.00	0.41	0.32
query25	0.12	0.05	0.04
query26	0.73	0.21	0.16
query27	0.04	0.04	0.03
query28	3.50	0.89	0.52
query29	12.48	4.12	3.30
query30	0.27	0.16	0.15
query31	2.77	0.60	0.32
query32	3.21	0.58	0.48
query33	3.25	3.24	3.17
query34	15.58	4.23	3.54
query35	3.58	3.55	3.53
query36	0.57	0.47	0.45
query37	0.08	0.07	0.07
query38	0.06	0.04	0.04
query39	0.03	0.03	0.03
query40	0.18	0.16	0.16
query41	0.08	0.04	0.03
query42	0.05	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.71 s
Total hot run time: 25.13 s

@starocean999
starocean999 marked this pull request as ready for review July 23, 2026 01:45
@starocean999 starocean999 changed the title [fix](preagg)Disable pre-aggregation for multi-child aggregate functions [fix](preagg) Enable pre-aggregation for multi-argument aggregate functions with all-key distinct inputs Jul 24, 2026
@starocean999

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review summary

Requested changes for three P1 correctness regressions in aggregate-key preaggregation classification. The direct count(DISTINCT k6, v7) case is now OFF and direct all-key count(DISTINCT k6, k5) is ON, but adjacent legal plans can still receive ON and observe partial storage rows.

Critical checkpoint conclusions:

  • Goal and correctness: not fully achieved; three wrong-result paths are described inline.
  • Scope and minimality: the patch is small and readable, but the input-slot-set criterion is broader than the supported argument semantics and assumes every slot was classified.
  • Parallel paths and performance: both SetPreAggStatus registrations are affected. The safe deterministic all-key optimization is preserved; the unsafe ON decisions change correctness rather than only performance. Planning cost is negligible.
  • Tests and results: the added EXPLAIN checks cover the direct mixed/all-key boundaries, but not an IF/CASE first argument with a later value column, a volatile key expression, or a multi-project derived alias. The weak OFF-reason assertion is already covered by the existing thread. No local build or test was run because the review contract prohibits it; current style, license, and basic checks are green, with no targeted regression result at this head.
  • Compatibility, persistence, transactions, writes, FE/BE protocol, concurrency, lifecycle, configuration, and memory: no independent issue applies.
  • Error handling and observability: the existing EXPLAIN OFF reason is sufficient; no new log or metric is needed.
  • Existing review context: the direct all-key discussion was deduplicated.
  • User focus: none was specified.

Review completion: every changed file and candidate was swept; three full/risk-focused rounds converged with NO_NEW_VALUABLE_FINDINGS on this final ledger and comment set.

Validation: static review only; builds and tests were prohibited by the governing review prompt.

@starocean999

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes

The all-key multi-argument DISTINCT path and the current-head fixes for the existing threads are directionally correct, but this review found four additional issues: three correctness/planning failures and one planner-complexity regression. Because M-004 was discovered in the third and final allowed round, the mandated convergence protocol ended without every reviewer returning NO_NEW_VALUABLE_FINDINGS; this review is therefore incomplete under the automation contract, although every candidate discovered so far has a verified disposition and is included below.

Critical checkpoint conclusions:

  • Goal and data correctness: the direct all-key multi-column DISTINCT case is enabled and the mixed-value cases are rejected, but volatile expressions can still observe partial-row multiplicity (M-002), and mixed validation can enable one scan using another scan's SUM value (M-003).
  • Scope and clarity: the patch remains within preaggregation classification and its regression suite, but eager alias composition introduces both redundant map copying (M-001) and an uncaught expression-limit failure (M-004).
  • Concurrency and lifecycle: no concurrent shared state, lock ordering, asynchronous lifecycle, or static-initialization concern is introduced; the context is per rewrite.
  • Configuration and compatibility: no new configuration, FE/BE protocol, storage-format, rolling-upgrade, or persisted-state change is present. M-004 incorrectly bypasses the existing expression width/depth limits rather than introducing a new setting.
  • Parallel paths and conditions: value-only, key-only, mixed IF/CASE, multi-argument DISTINCT, sibling projects, and join paths were traced. The filter/join/grouping and other-table volatile paths are not covered by the new local guard, and mixed key/value ownership is not scan-relative.
  • Tests and results: the added EXPLAIN cases cover direct all-key/mixed DISTINCT, local volatile aggregate arguments/conditions, and the repaired sibling alias case. Coverage is still missing for volatile filter/join/grouping and other-table aggregates, cross-scan foreign-SUM multiplicity, and over-limit nested project composition. This runner's review contract prohibited builds/tests; validation was static only, and thirdparty/installed/bin/protoc is absent.
  • Performance: M-001 adds O(N^2) prefix copying across projected branches; no other material hot-path regression was substantiated.
  • Observability: existing PREAGGREGATION ON/OFF reasons are sufficient for this rule; no new metric or logging requirement was identified.
  • Transactions, persistence, data writes, and FE/BE variable propagation: not applicable to these planner-only changes.
  • User focus: no additional user-provided focus was supplied.

@starocean999

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jul 30th, 2026 7:58 AM.
Workflow run: https://github.com/apache/doris/actions/runs/30079106736

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants